Estimation of Entropy and Mutual Information
نویسندگان
چکیده
منابع مشابه
Estimation of Entropy and Mutual Information
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander’s method of sieves and places no assumptions on the underlying probabilit...
متن کاملGeometric k-nearest neighbor estimation of entropy and mutual information
Nonparametric estimation of mutual information is used in a wide range of scientific problems to quantify dependence between variables. The k-nearest neighbor (knn) methods are consistent, and therefore expected to work well for a large sample size. These methods use geometrically regular local volume elements. This practice allows maximum localization of the volume elements, but can also induc...
متن کاملEfficient Entropy Estimation for Mutual Information Analysis Using B-Splines
The Correlation Power Analysis (CPA) is probably the most used side-channel attack because it seems to fit the power model of most standard CMOS devices and is very efficiently computed. However, the Pearson correlation coefficient used in the CPA measures only linear statistical dependences where the Mutual Information (MI) takes into account both linear and nonlinear dependences. Even if ther...
متن کاملInformation Theory 4.1 Entropy and Mutual Information
Neural encoding and decoding focus on the question: " What does the response of a neuron tell us about a stimulus ". In this chapter we consider a related but different question: " How much does the neural response tell us about a stimulus ". The techniques of information theory allow us to answer this question in a quantitative manner. Furthermore, we can use them to ask what forms of neural r...
متن کاملMutual information challenges entropy bounds
We consider some formulations of the entropy bounds at the semiclassical level. The entropy S(V ) localized in a region V is divergent in quantum field theory (QFT). Instead of it we focus on the mutual information I(V,W ) = S(V ) + S(W ) − S(V ∪W ) between two different non-intersecting sets V and W . This is a low energy quantity, independent of the regularization scheme. In addition, the mut...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Neural Computation
سال: 2003
ISSN: 0899-7667,1530-888X
DOI: 10.1162/089976603321780272